Search for GeV-scale sterile neutrinos responsible for active 
neutrino oscillations and baryon asymmetry of the Universe 



S.N. Gninenko 1 , D.S. Gorbunov 1 , and M.E. Shaposhnikov 2 

1 Institute for Nuclear Research of Russian Academy of Sciences, 117312 Moscow, Russia 
2 Institut de Theorie des Phenomenes Physiques, EPFL, CH-1015 Lausanne, Switzerland 

Abstract 

Standard model of particle physics fails to explain neutrino oscillations, dark matter 
and baryon asymmetry of the Universe. All these problems can be solved with three sterile 
neutrinos added to the model. Quite remarkably, if sterile neutrino masses are well below 
the electroweak scale, this modification — Neutrino Minimal Standard Model (z/MSM) 
- can be tested experimentally. We discuss a new experiment on search for decays of 
GeV-scale sterile neutrinos, which in the framework of the i/MSM are responsible for the 
matter-antimatter asymmetry generation and for the active neutrino masses. If lighter 
than 2 GeV, these particles can be produced in decays of charm mesons generated by 
high energy protons in a target of a proton beam-dump experiment, and subsequently 
decay into light SM particles. In order to fully explore this sector of the vMSM, the 
new experiment requires data obtained with at least 10 20 incident protons on target 
(achievable at CERN SPS in the future), and a big volume detector constructed from a 
large amount of identical single modules, with a total sterile neutrino decay length of few 
kilometers. The preliminary feasibility study for the proposed experiment shows that it 
has sensitivity which may either lead to the discovery of new particles below the Fermi 
scale — right-handed partners of neutrinos — or rule out seesaw sterile neutrinos with 
masses below 2 GeV. 

1 Introduction 

The discovery of neutrino oscillations provides an undisputed signal that the Standard Model 
(SM) of elementary particles is not complete. However, what kind of new physics it brings 
to us remains still unclear: we do not know yet the properties of new particles which are 
believed to be behind this phenomenon. An attractive possibility is the extension of the SM 
by three right-handed neutrinos, making the leptonic sector similar to the quark one, see 
Fig.l. 

The masses M and Yukawa couplings Y of new leptons remain largely unknown. Basically, 
M can have any value between zero (corresponding to Dirac neutrinos) and 10 16 GeV, whereas 
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Figure 1: Particle content of the SM and its minimal extension in neutrino sector. In the SM 
(left) the right-handed partners of neutrinos are absent. In the z/MSM (right) all fermions 
have both left and right-handed components. 



Y can vary from 10 -13 (Dirac neutrino case) to ~ 7r (the onset of the strong coupling). The 
admitted region is sketched in Fig. 2. 

Independently on their mass, the new Majorana leptons can explain oscillations of active 
neutrinos. So, an extra input is needed to fix their mass range. It can be provided by the 
LHC experiments. 

Suppose that the resonance found at the LHC by Atlas and CMS in the region 125 — 126 
GeV is indeed the Higgs boson of the Standard Model. This number is remarkably close to 
the lower limit on the Higgs mass coming from the requirement of the absolute stability of 
the electroweak vacuum and from Higgs inflation, and to prediction of the Higgs mass from 
asymptotic safety of the Standard Model (see detailed discussion in [1] and in a proposal 
submitted to European High Energy Strategy Group by Bezrukov et al. [2]). The existence 
of the Higgs boson with this particular mass tells that the Standard Model vacuum is stable 
or metastable with the life-time exceeding that of the Universe. The SM in this case is a 
valid effective field theory up to the Planck scale, and no new physics is required above the 
Fermi scale from this point of view. Suppose also that the LHC finds no new particle and no 
deviations from the Standard Model. In this case the "naturalness paradigm", leading the 
theoretical research over the last few decades will be much less attractive, as the proposals for 
new physics stabilizing the electroweak scale by existence of new particles in the TeV region 
and based on low energy supersymmetry, technicolor or large extra dimensions would require 
severe fine-tunings. 

The solution of the hierarchy problem does not require in fact the presence of new particles 
or new physics above the Fermi scale. Moreover, the absence of new particles between the 
electroweak and Planck scales, supplemented by extra symmetries (such as the scale invari- 
ance) may itself be used as an instrument towards a solution of the problem of stability of 
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Figure 2: The admitted values of the Yukawa couplings of sterile neutrinos as a function of 
their seesaw Majorana masses. 

the Higgs mass against radiative corrections (for detailed arguments see [3, 4, 5]). 

Even regardless the hierarchy problem, it is clear that the Standard Model of elementary 
particles is not complete. It is in conflict with several observations. These are non-zero 
neutrino masses and oscillations discussed above, the excess of matter over antimatter in the 
Universe, and the presence of non-baryonic dark matter. Any model of physics beyond the 
SM (BSM) should be able to deal with the experimental troubles of the SM. 

Guided by the arguments steaming from alternative solutions to the hierarchy problem it 
is natural to ask whether the observational problems of the SM can be solved by new physics 
below the Fermi scale. And the answer is affirmative: an economic way to handle in a unified 
way the problems of neutrino masses, dark matter and baryon asymmetry of the Universe 
is to add to the SM three Majorana singlet fermions with masses roughly of the order of 
masses of known quarks and leptons. This theory is called the i^MSM, for "Neutrino Minimal 
Standard Model" (for a review see [8]). The lightest of the three new leptons is expected 
to have a mass from 1 keV to 50 keV and plays the role of the dark matter particle (see 
detailed discussion in [8] and in a proposal submitted to European High Energy Strategy 
Group by Boyarsky et al. [9]). Two other neutral fermions are responsible for giving masses 
to ordinary neutrinos via the see-saw mechanism at the electroweak scale and to creation of 
the baryon asymmetry of the Universe. The masses of these particles and their couplings 
to ordinary leptons are constrained by particle physics experiments and cosmology. Two 
leptons should be almost degenerate, forming thus nearly Dirac fermion (this is coming from 
the requirement of successful baryogenesis) . For comparison, we show in Fig. 3 the summary 
of different possibilities for the masses of Majorana leptons. 

A lot of experimental efforts were devoted to the direct search of Majorana neutral leptons 
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Figure 3: This table shows whether a given choice of the mass of sterile neutrinos can explain 
neutrino masses and oscillations, accommodate eV neutrino anomalies, lead to baryogenesis, 
provide a dark matter candidate, ensure the stability of the Higgs mass against radiative 
corrections, and be directly searched at some experiments. 

in the past [12]- [20]. No new particles were found, but several constraints on their mixing 
angles with ordinary leptons were derived. The interest to these searches declined consider- 
ably at nineties, most probably due to the theoretical prejudice that the masses of Majorana 
leptons should be associated with the Grand Unified scale ~ 10 16 GeV [21]- [24], making their 
direct search impossible. As a result, no dedicated searches of relatively light neutral leptons 
were performed in the last decade, with an exception of several experiments [17]-[19] related 
to the so-called Karmen anomaly [25]. Today, the reason for the negative result became clear: 
in the domain of parameters accessible to most of these experiments the neutral leptons in- 
teract too strongly to produce the baryon asymmetry of the Universe, not satisfying one of 
the Sakharov conditions for baryogenesis. 

The recent theory developments provide a guideline for the required experimental sensitiv- 
ity [26], which was absent at the time when the early experiments were made. The aim of 
the present work is to discuss the experimental signatures of the existence of these particles 
and estimate the parameters of the experiment of beam-target type required to fully explore 
the model parameter space if sterile neutrinos are lighter than D-mesons (see a proposal 
submitted to European High Energy Strategy Group by Gorbunov and Shaposhnikov [11]). 

The paper is organized as follows. In Section 2 we give the essential features of the model 
and discuss the constraints on new particles. Section 3 is devoted to general description of 
experiments on searches for sterile neutrinos of masses in GeV range. In Section 4 we present 
an estimate of sensitivity required to test the model and suggest the experimental setup to 
accomplish the ultimate task. Section 5 contains conclusions. 
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2 The model and constraints on properties of new particles 



The z^MSM is described by the most general renormalizable Lagrangian of all the SM par- 
ticles and 3 singlet fermions. For the purpose of the present discussion we take away from 
it the lightest singlet fermion N± (the "dark matter sterile neutrino"), which is coupled ex- 
tremely weakly to the ordinary leptons. In addition, we take N 2 and N 3 degenerate in mass, 
M 2 = M3 = M. The approximate equality of the masses of N 2:3 comes from requirement of 
resonance production of the baryon asymmetry of the Universe. Then a convenient param- 
eterization of the mass term and the interaction of iV's with the leptons of SM is given in 
terms of three parameters (M, e, rj): 



-^singlet 



^ v / Lvee"? 



H - MN 2 N 3 + h.c. , (1) 
where L 2 and L 3 are the combinations of the left-handed doublets L e , L M and L T , 

L 2 = ^2 XaLa ' L 3 = ^2 VaLa , (2) 

a a 

with J2 a \ x a\ 2 = "Yla \Vc\ 2 = 1- ^ n ec l- (-0 v = ^46 GeV is the vacuum expectation value of 
the Higgs field H, Hi = £ijH*, rrii are the active neutrino masses. Note that one of them is 
negligibly small in comparison with others in the z/MSM [8]), so that J2 m i = Km atm with 
niatm = V\^ m atm\ — e ^ being the atmospheric neutrino mass difference, k = 1 (2) 
for normal (inverted) hierarchy of neutrino masses. Mixing parameters x a and y a can be 
expressed through the parameters of the active neutrino mixing matrix (explicit relations 
can be found in [10]). The parameter e (by definition, e < 1) and the CP-breaking phase r/ 
cannot be determined by neutrino oscillation experiments. 

If the value of mass parameter M is fixed, smaller e yields stronger interactions of singlet 
fermions to the SM leptons. This leads to equilibration of these particles in the early Universe 
above the electroweak temperatures, and, therefore, to erasing of the baryon asymmetry. In 
other words, the mixing angle U 2 between neutral leptons and active neutrinos must be small, 
explaining why these new particles have not been seen previously. For small e, 

U 2 = ^ . (3) 
4Me V ; 

The region, where baryogenesis is possible in (U 2 , M) plane is shown in Fig. 4. We also plot 
there the exclusion regions coming from different experiments such as CHARM [14], NuTeV 
[17], and CERN PS191 experiment [15, 16] (see also discussion of different experiments in 
[27, 6]). Only CERN PS191 significantly entered into the cosmologically interesting part 
of the parameter space of the z^MSM for sterile neutrinos lighter than kaon. In Fig. 5 we 
present the expected lifetime of the singlet fermions in an experimentally interesting region 
M < 1.8 GeV. 
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Figure 4: The allowed region of parameters of sterile neutrinos, responsible for neutrino oscil- 
lations (region above the dotted "see-saw" line) and for baryo/leptogenesis (region between 
two black solid lines). Sterile neutrinos with the parameters in the shaded region to the left 
of the "BBN" line would spoil predictions of primordial nucleosynthesis. Accelerator exper- 
iments, searching for heavy neutral leptons exclude regions above green lines. Left panel: 
restrictions for normal hierarchy, Right panel: inverted hierarchy. Adopted from Ref. [7]. 
The region above the red curve can be probed with a single section of the detector of 
length /|| ~ 100 m, height 5 m and width l± ~ 5 m, placed at a distance of about hundred 
meters; see the text for details. 

As we see, the mass of these particles is limited from below by current experiments and 
cosmological considerations by ~ 100 MeV, while no known solid upper bound, better than 
the electroweak scale, can be applied. At the same time, various considerations indicate that 
their mass may be in 0(1) GeV region [10]. 

3 General considerations for the experiment 

The most efficient mechanism of sterile neutrino production is associated with weak decays 
of heavy mesons and baryons, see left panel of Fig. 6 for examples of relevant two- and 
three-body decays. Therefore, from experimental point of view, the distinct mass ranges are 
associated with the masses of mesons in which sterile neutrinos can be created: below 500 
MeV (K meson), between 500 MeV and 2 GeV (D-mesons), between 2 and 5 GeV (B-mesons), 
and above 5 GeV. We will concentrate here on the masses below the charm threshold, the 
higher values will be discussed shortly in Section 5. 

The mechanism works as follows. Heavy mesons (and baryons) can be produced by en- 
ergetic protons scattering off the target material. As a reference machine we take CERN 
SPS with beam energy of 400 GeV. Here the relevant heavy hadrons contain charm as the 
heaviest quark: D, D s , A c , D* , . . .With typical weak lifetime (in the rest frame) of about 
10" 10 s these mesons mostly decay before further interaction in the target, no matter how 
thick it is. Sterile-active neutrino mixing gives rise to sterile neutrino production in weak 



6 




0.2 0.5 1.0 2.0 5.0 

M [GeV] 

Figure 5: Constraints on the sterile neutrino lifetime tn coming from the baryon asymmetry 
of the Universe (blue solid lines: ll BAU N" refers to the normal hierarchy, ll BAU I" refers to 
the inverted hierarchy), from the seesaw mechanism (blue solid lines: 11 Seesaw N" refers to 
normal hierarchy, "Seesaw I" refers to the inverted hierarchy) and big bang nucleosynthesis 
(black line: "BBN"). The allowed region of parameter space is in white. Limits from direct 
searches presented in Fig. 4 are not outlined. 

decays of the heavy mesons 1 . For charmed mesons typical branching ratios expected for the 
i^MSM parameter space are at the level of [26] 

Br (D -> N) ~ 1(T 8 - 1(T 12 (4) 

(referring to upper and lower limits in Fig. 4, respectively). 

These sterile neutrinos further weakly decay to the SM particles due to mixing with active 
neutrinos. In the z^MSM mixing angles are small, see Fig. 4, hence the sterile neutrinos live 
much longer (see Fig. 5) as compared to that of the weakly decaying SM particles of similar 
masses. Before the decay (and with account of 7-factor) relativistic sterile neutrinos would 
cover quite a large distance significantly exceeding ten kilometers. Hence, sterile neutrino 
decays into SM particles due to mixing with active neutrino can be searched for in the near 
detector, see Fig. 7. Two examples of the interesting decay modes are presented on right 
panel of Fig. 6. More examples and typical patterns of sterile neutrino decay branching 
ratios expected within the i^MSM can be found in ref. [26]. 

Let us first make rough estimates of length-scale of the detector needed to fully explore the 
z^MSM parameter space for sterile neutrino masses in the interesting range Mn ~ 0.5 — 2 GeV, 
where sterile neutrinos are dominantly produced in charmed hadron decays. Let Pt and Pl 
be average transverse and longitudinal momenta of produced sterile neutrinos. We start with 

x Note that searches for those heavy hadron decays is one of the possible strategy of exploring the z^MSM 
parameter space in full, which at the present level of experimental technique seems unrealistic, see discussion 
in Section 5. 
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Figure 6: Left panel: Feynman diagrams of meson decays to heavy sterile neutrinos. Right 
panel: Feynman diagrams of sterile neutrino decays. 



beam 




Figure 7: Sketch of typical beam-target experiment on searches for sterile neutrino decays. 

detector of a special geometry designed to cover entirely the solid angle ~ nPj^/P? around the 
beam axis. Then all the sterile neutrinos travel through the fiducial volume of the detector 
which has a conical form. For 400 GeV proton beam operating at SPS the charm production 
cross section a c was measured by LEBC/EHS collaboration revealing a c ~ 30 fib [29]. Thus 
for iVp t = 10 20 protons incident on the target per year one expects 

— AW ~ 0.75 x 10 17 

(Tp 

charmed hadrons (o~ p ~ 40 mb [30] is the total proton cross section at SPS energy). With 
charm branching ratio to sterile neutrino given above (4) one expects about 

N N ~ 0.75 x (10 5 - 10 9 ) 

sterile neutrinos crossing the detector each year of operating. 

Then, for reference sterile neutrino mass of 1 GeV one estimates the decay length of 10 2 — 
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10 5 km (see Fig. 5) and hence 

Nff c ~ 1 - 10 7 

decay events per year for 1 km-length detector. Note, that with Pt/Pl ~ 1 GeV/40 GeV [29] 
and heavy neutrino following the parent meson travel direction, the detector at a distance of 
1 km has to cover the area of about 25 mx25 m to accomplish the task. 

Considering a more realistic setup, where detector is placed on the Earth surface and its 
height is of about 5 m (cf. the size of CHARM detector [14] ) one concludes that the detector 
has to be about 5 times longer to achieve the same sensitivity to the sterile neutrinos as in 
the previous conical configuration. 

To register all the expected sterile neutrino decays it seems appropriate to install many 
detectors of a reasonable 5mx5mxl00m size rather than one large detector. These small 
detectors may operate separately by independent groups of experimentalists but together 
cover the same volume as the large detector, see Fig. 8. 

E/HCAL 

Beam dump 



P, 400 GeV T fX L— 

- ^ - : 




Figure 8: Schematic illustration of a proton beam dump experiment on search for D s — > 
fiN, N —7- [iir decay chain: charm mesons D s generated by the proton beam in the target (T) 
produce a flux of high energy iV's through the U^n mixing in the decay D s — > [iVa, which 
penetrate the downstream shielding and decay into fin pair in a neutrino decay volume (DV) . 
The same setup can be used to search for the process N — > [iev. See text. 



4 Preliminary study for the feasibility of the proposed exper- 
iment 

The searches for Majorana leptons were undertaken in the past at PS191, BEBC, CHARM, 
NuTeV and led to a negative result, not surprising in view of the cosmological constraints 
on the properties of sterile neutrinos, that were derived later on. The decays of the v MSM 
sterile neutrinos are very rare events, and their observation presents a challenge for the 
detector design and performance. The sensitivity of the search has [t/^jvl 4 dependence on 
the mixing stength, and in order to increase it substantially as compared to the previous 
searches, one has to use: 
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• higher intensity of the proton beam and larger amounts of pot's. 

• shorter distance to the proton target 

• larger N decay fiducial volume 

• longer time of data accumulation 

• better efficiency of events detection, reconstruction, and background rejection, etc... 

To compromise the cost of the detector and its performance, we focus mainly on discussions of 
the experimental setup to search for the iV decays in a detector volume filled with a (diluted) 
gas, not in vacuum. The setup with vacuum requirements would provide possibly better 
sensitivity, but is more complicated and costly. 

The main components of the experimental setup to search for the N decays of sterile 
neutrino are schematically illustrated in Fig. 8. The full detector is composed of single 
identical modules. Each module consists of a decay volume tank (DV) with diameter of 
about 5-7 meters, and length of 20-30 meters. To minimize cost and amount of passive 
material, the decay tank could be, e.g. a rubber ballon filled with helium. Each DV is 
followed by a high precision tracker, which could be a layers of straw tube (ST), see e.g. 
[36]. The entire detector is assembled from ~ 50-100 such modules and is followed by a 
calorimeter (E/HCAL) which has the electromagnetic and hadronic parts. The detector is 
protected against charged secondaries from neutrino interactions in the dump by a Veto (V) . 

The design of a single module is similar to that one used in the PS191, or CHARM exper- 
iments, that searched for decays A — > ue + e~, /itt, fteu, .. of heavy neutrinos in the A mass 
range from 10 MeV to 1.8 GeV originated from decays tt, K and charmed D mesons decays 
[14, 15, 16]. The later experiment, specifically designed to search for neutrino decays in a 
high-energy neutrino beam, was performed by using 400 GeV protons from the CERN Super 
Proton Synchrotron (SPS) with the total number of 2.4 x 10 18 protons on (Cu) target (pot). 
The CHARM decay detector (DD), located at the distance of 480 m from the target, consists 
of decay volume of 3 x 3 x 35 m 3 , three chambers modules located inside the volume to detect 
charged tracks and followed by a calorimeter. The decay volume was essentially an empty 
region to substantially reduced the number of ordinary neutrino interactions. The signature 
of the heavy neutrino decay — > ve + e~ were events originating in the decay region at a 
small angle with respect to the neutrino beam axis with one or two separate electromagnetic 
showers in the calorimeter [14]. No such events were observed and limits were established on 
the v e ^ — Vh mixing strength as a function of the mass. 

4.1 Production and decay of sterile neutrinos 

If the decays D s , D — > fiN exist, one expects a flux of high energy A's from the SPS target, 
since neutral mesons D s and D are abundantly produced in the forward direction by high 
energy protons in the target. If A is a relatively long-lived particle, this flux would penetrate 
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the downstream shielding without significant attenuation and would be observed in the pro- 
posed detector via the N — > fin, fiev decay into a high energy fin or fie pair, as schematically 
illustrated in Fig. 7. The experimental signature of the sterile neutrino decays is an observa- 
tion of either fin or fie pairs originated from a common vertex located in empty space. This 
signature is clean and the signal events are expected to be selected in the detector with a 
small background. The occurrence of N — > fin, fiev decays would appear as an excess of fin 
or fie pairs in the detector above those expected from standard neutrino interactions. Up to 
a N mass of < 2 GeV , the mass difference between the D s and the fi, A's can originate 
both from the v„ produced directly in the D s decay and from the v^ produced indirectly 
in the subsequent decay of r produced in D s decays. For a N mass below than < 1.8 GeV 
v^s produced in r decay can contribute. However these indirect TV's have a lower acceptance 
resulting in a smaller probability to observe the N decays in the detector. Therefore we focus 
mainly on directly produced iV's. 

As discussed previously, the mixing between the sterile neutrino and the muon neutrino 
results in the decay D s — > fiN, as illustrated in Fig. 8. The D s meson, which is normally 
decays into a fi and v^, might instead decay to a fi and a heavy neutrino N. For the mass 
interval m Vh < 2 GeV the chirality-flip is mostly due to sterile neutrino mass which results 
in 



T(D s ^fiN)^T(D s ^fi^\U, N \ 2 (^) . (5) 

In the SM, D s meson decays leptonically via annihilation of the c and s quarks through a 
virtual W + . The decay rate of this process is given by 

.2 



2 



F(D S -+ lu) = ^f D mfM D h - —L- ) \V CS \ 2 , ((>) 



where the Md 3 is the D s meson mass, mi is the mass of the charged lepton, /|, is the 
decay constant, Gf is the Fermi constant, and V cs is a Cabibbo-Kobayashi-Maskawa matrix 
element which value equals 0.97334 [35]. The decay rate (6) is suppressed by the lepton 
mass squared, since the very leptonic decay is due to chirality-flip. As follows from Eq.(5), 
in neutrino scattering experiments the ./V decay signal rate is proportional to oc IC^nI 4 (the 
mixing |J7u/i| 2 appears twice, through the heavy neutrino production, and through the its 
decays in the detector) and, thus is more suppressed. The branching ratio of a particular 
decay mode, e.g. N fin (or — > fiev) is given by 

BR(N fin) = = F(M N , |^| 2 ) , (7) 

1 tot 

where the function F(Mn, \U^n\ 2 ) is calculated in [26], and the total rate T to t is dominated 
by the N — > *&v decay channel. 

To make quantitative estimates, we performed simplified simulations of the N production 
in the inclusive reaction 

p + Be D s + X, D s fi + N ->■ fin, fiev (8) 
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with the emission of a GeV-scale sterile neutrino N subsequently decaying semi-leptonically 
or leptonically in the detector, as illustrated in Figure 7,8. In the simulations it is assumed, 
that the N is a long-lived particle and the flux of iV's penetrates the downstream shielding 
without significant attenuation. The decay N — > jjltt, \iev cannot be distinguished from the 
anti-neutrino decay N — > fin or N — > [lev and the obtained result therefore refers to the sum 
of all these decay modes. 

The flux of sterile neutrinos from decays of ZVs and D's produced in the target by primary 
protons can be expressed as follows: 



• W oc / " V(P+ ^ P ' (P) + X) 1W» - vN) f <fp aM , (9) 

J d PD s (D) 



where iVp t is the number of pot, a(p + N — > D S (D) + X) is the D S (D) meson production 
cross-section, Bv(D s (D) — > fiN) is the D S (D) — > fiN decay mode branching fraction [35], and 
/ is the decay phase space factor, respectively. 

To perform quantitative we used simulations of the process shown in Fig. 7 by taken into 
account the relative normalization of the yield of charmed meson species D s and D from 
the original publications. The invariant cross section of a charm meson production can be 
expressed as [35] 

E— = d3(T = S(J , (10) 
d 3 p pTdpTdydcf) 2irpTdpTdy 

where pr is the transverse momentum of the particle, y is its rapidity, and in the last equality 
integration over the full 2ir azimuthal angle (j) is performed. For the production cross sections 
of D s and D charm mesons we used the Bourquin-Gaillard (B-G) formula from Ref. [37]: 

Jp d 3 a(p + N^D S (D) + X) , 2 .12.3 , 5.13. 

E = a{ eV^2 ] e ^-yoM)f(PTl (ii) 



where 



fipl 



exp(-pT) Pt < 1 GeV 

exp(-l-23(pr-l)A/s) PT > 1 GeV 



with Y = y-max — y, being the rapidity, s is the Mandelstam variable in GeV 2 , and Et(pt) is 
the transverse energy (momentum) in GeV. The coefficient A in (11) is a normalization factor 
that was tuned to obtain the cross sections of the charm production in pp collisions at 400 
GeV. The spectra of v^s produced in the Be target by primary protons could be calculated by 
using the approach reported in ref. [38] (see also [39]). However, the Bourquin-Gaillard (B- 
G) approach gives the parametric form of (10) for the production of many different hadrons, 
including charm, which is in good agreement with data over the full phase-space in high-energy 
hadronic collisions and with a precision which is satisfactory at the level of our accuracy. The 
total D s and D production cross sections in p-Be collisions were calculated from its linear 
extrapolation to the target atomic number. The contribution of protons not interacting in 
the Be target and interacting in the SPS beam dump was not taken into account. 
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Figure 9: Two dimensional distribution of flux for 1 GeV sterile neutrino as a function of Qn, 
the angle between the proton beam axis and momentum of N, and its energy calculated for 
400 GeV proton incident on Be target. The N's are produced through the decay D s — > /j,N 
assuming mixing strength \U^n\ 2 = 1- 

The calculated fluxes and energy distributions of D S ,D produced in the Be target were 
used to predict the flux of TV's, as a function of its mass. An example of 2D-flux distribution 
calculated for 1 GeV sterile neutrino as a function of Bat, the angle between the proton 
beam axis and momentum of N, and the N energy for 400 GeV proton incident on target 
is shown in Fig. 9. In these calculation the N's are produced through the decay D s — > [iN 
assuming mixing strength IU^n] 2 = 1- One can see, that for detector located at the distance 
~ 100 m from the target its fiducial area for efficient N detection should be of the order 
S ~ vrL 2 9 2 -300 m 2 . 

4.2 Background 

There are several expected sources of background in the experiment, which come from 
• neutrino interactions in hellium 
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• neutrino interactions in surrounding materials 

• cosmics 

• accidentals 

• punch-through energetic neutral particles, such as K® or n 

Because of the high number of pot's and many possible sources of background the detail study 
for the feasibility of the proposed search would require the simulation of a very large number of 
events resulting in a prohibitively large amount of computer time. Consequently, only small 
fraction of the required statistics for u fl CC(NC) inelastic reactions and other background 
components, e.g. such as coherent and quasi-elastic reactions, etc., were considered for 
preliminary estimate and combined with numerical calculations. 

The estimate shows that the dominant background is expected from the primary in- 
teractions in passive material, which correspond roug hly to ~ 10 6 z^CC/lO 20 pot events 
per 1 ton of matter. The largest contribution to background is expected from neutrino in- 
teractions yielding a single fiir pair with little hadronic activity in the final state. Neutrino 
interactions that occur in the surrounding material, e.g. upstream of the DV fiducial volume 
or in material of the decay tanks, also may produce an isolated fin pairs if the accompa- 
nying particles are not detected. These background events from surrounding material could 
be rejected with an effective method developed in Ref.[19], by introducing the collinearity 
variable 1 — cos ©, where is the angle between, e.g. the /U7r pair momentum direction and 
the neutrino beam axis. The main idea is that if the /X7r pair originates from the decay chain 
D s —7- fiN, N —7- fiir, at high energies the fin pair momentum extrapolated back should point 
to the D s production target. An example of the distribution of the variable 1 — cos0 ob- 
tained in the NOMAD experiment searching for the decay ./V — > e + e~i / , is shown in Figure 5 
of Ref.[19]. One can see that potentially one could measure the [m pair momentum direction 
with precision 1 — cos O ~ 1CT 5 . By using this variable, two independent techniques could be 
used for the background estimation in the signal region. The first is obviously based on the 
MC. The second method relies on the data themselves. By extrapolating the observed events 
to the signal region with the shape of the fully simulated MC events a background estimate 
can be obtained from the data. Using this technique would results in an additional rejection 
of background events. 

4.3 Sensitivity estimate 

For a given flux cI&(Mn, En, N pot )/dEN of A's the expected number of N — > ^ir, neu decays 
occurring within the fiducial length L of the detector located at a distance L' from the 
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Figure 10: The expected 90% C.L. upper limits on the branching ratio Br(D s —> 
fiN)Br(N —> fiir) versus estimated for the proposed experiment in the background free 
case. The numbers near the curves indicate the corresponding values of sterile neutrino 
masses. 



neutrino target is given by 



N N ^ itiev = Br(D s -> fj,N)Br(N -»• (j,ir) J 



. ex ( L ' M N \U f LM N \ 

V P N T N J I V P N T N ) 



dE N 

(SdE N , (12) 



where En,Pn, and tm are the N energy, momentum and the lifetime at rest, respectively, 
and C(— 50%) is the fiir (fie) pairs reconstruction efficiency. The acceptance 5 was calculated 
by tracing JV's produced in the Be-target to the detector taking the relevant momentum and 
angular distributions into account. In this estimate the average momentum is < pn >— 45 
GeV , and S ~ 50%. 

In the case of no signal observation, the background free experiment would result in strin- 
gent constraints on existence of the sub-GeV sector of the z^MSM. An example of the exclusion 
region calculated in the (Br(D s — > fiN)Br(N — > //7r);rjv) plane for ~ 10 20 pot and the de- 
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tector assembled from ~ 10 2 modules, assuming N^^^^ ev < 2.3 at 90% CL, is shown in 
Fig. 10. For the mass range 0.5 < < 2 GeV these limits corresponds to the bounds on 
tne mixing strength which are in the range 10~ 6 < \U^n\ 2 < 10~ 10 and exclude seesaw sterile 
neutrinos with masses below 2 GeV. 

Note, that the uncertainty for the D s production at low pt does not significantly affect the 
sensitivity estimate. For example, the variation of the D s yield in (12) by 30% results in the 
corresponding variation of the limits on mixing strength U^n of order 5%. This is because 
the sensitivity of the search is proportional to the |?7 M 7v| 4 . Indeed, in (12) the branching 
fraction of (6) and the decay rate T(N — > /j,ir , /lev) of (5) both are proportional to (l^vl 2 . 

5 Conclusions 

The energy scale of new physics is not known. If it exists at energies above the Fermi scale 
(examples include supersymmetry, large or warped extra dimensions, models with dynamical 
electroweak symmetry breaking), the search for new particles can be carried out in direct 
experiments, such as ATLAS and CMS at LHC. In addition, new hypothetical heavy particles 
inevitably appear as virtual states, leading to different rare processes, absent in the SM. 
These effects can be found at experiments such as LHCb and are competitive with the direct 
searches. If no new physics is found at the LHC, this may indicate the absence of new physics 
all the way between the Fermi and Planck scales. 

Quite paradoxically, the largely unexplored up to now domain of energies where the new 
physics can be hidden is related to physics below the Fermi scale. If the new particles are light 
and weakly interacting (as in the z/MSM, discussed here) then the search for rare processes 
is superior to high energy experiments. It provides a unique possibility for discovery of new 
physics, not accessible by any of the LHC experiments. 

The present proposal deals with the searches of new relatively light Majorana leptons which 
"give" the masses to ordinary neutrinos and produce baryon asymmetry of the Universe. We 
discussed here the parameters of the beam-target experiment which can explore completely 
the mass range below 2 GeV. Due to the weak coupling of these particles to other fermions 
these experiments are very challenging and require construction of a kilometer scale detector. 
However, an experiment at much smaller scales, with 100 meters detectors, would be able to 
explore the dominant fraction of the parameter space. 

Below we describe shortly the prospect for other experiments and higher range of sterile 
neutrino masses. 

We start with remark that signal statistics in beam-target experiment proposed in this 
paper is determined by the fourth power of sterile-active neutrino mixing, U 2 xU 2 , since both 
sterile neutrino production in meson decays and rate of subsequent neutrino decay within a 
detector are proportional to U 2 . Experiments on searches of meson decays to sterile neutrinos 
requires (potentially) less statistics of mesons, since the signal statistics are proportional only 
to the second power of sterile- active neutrino mixing U 2 . 
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With expected for z^MSM branching ratios (4) the relevant experiment has to collect not less 
than 10 12 hadrons in order to fully explore the entire part of parameter space at Mjy < 2 GeV. 
This statistics is far above what has been obtained at previous c- and 6-factories (CLEO(-c), 
BaBar, Belle) or at proposed future or (Novosibirsk) and S-factories (Tsukuba, Frascati), 
where total number of D-meson is planned at the level of 10 9 , see e.g. [32]. Only at LHC the 
.D-meson production rate is enough to perform this search, however, limits from data taking 
rate, QCD-background, vertex resolution, particle misidentification etc, prevent measurement 
of D- meson branching ratios at the required level (4) . 

As we have already mentioned, there are no solid-motivated upper limits on sterile neu- 
trino mass of sub-electroweak scale. Sterile neutrinos can be heavier than D-meson, and then 
charmed hadrons are out of use. If sterile neutrinos are lighter than 5 GeV, they can be pro- 
duced in beauty hadron decays. This mass range is favorable [7] for a particular mechanism 
of dark matter production (resonant oscillations in the primordial plasma with lepton asym- 
metry) in the early Universe. There are other options for dark matter production, which are 
insensitive to the 1 GeV scale sterile neutrinos, see e.g. [33, 34]. Typical branching ratios of 
5-mesons to sterile neutrino expected within i/MSM are [26] by one-two orders of magnitude 
smaller, than those of Z)-mesons, as both upper and lower bounds on mixing U 2 decrease with 
increase of sterile neutrino masses, see Fig. 4. Proposed upgrade of LHCb [31] will be able 
of entering the interesting region of z^MSM parameter space by searching for 6-meson decays 
into sterile neutrinos, but can not cover the entire region. Once can further discuss an idea of 
using LHC-beam (4TeV, 7TeV, higher) for a beam-target experiment of the kind proposed 
in this paper, in order to fully explore the zaMSM parameter space, if sterile neutrino mass 
belongs to the the interval 2-5 GeV. 

The preliminary analysis shows that the quoted sensitivity could be obtained with a proton 
beam and a detector optimized for several its properties. Namely, i) the primary beam 
intensity and the total number of pot's ii) the fiducial decay volume of the detector, hi) the 
composition of the material of decay tank and surrounding material and iv) the efficiency and 
precision of the tracking system, v) the E/HCAL granularity, energy resolution and particle 
identification, and vi) different sources of background a are of importance. 

As far as the cost concerned, each module may be relatively cheap, empty-space with simple 
tracker system either inside or behind the decay volume, and the calorimeter at the far end. 
Its design may repeat the design of the experiment on precision neutrino measurements and 
searches for sterile neutrino decays at FNAL [36]. The detector of the 100 length has a 
sensitivity to U 2 x U 2 worth by a factor only 50. Thus even one (the first) section will be 
able to deepen considerably into the parameter range interesting for cosmology, see Figs. 4. 

The work was supported in part by the grant of the President of the Russian Federation NS- 
5590.2012.2, by the SCOPES program (D.G. and M.S.) and by the RFBR grant ll-02-01528a 
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